Video frame segmentation is the process for breaking a video frame into a background scene and foreground objects. Typically, the video frame segmentation process involves the construction of a background mosaic. A background mosaic generates a complete still image of the background from video frames within a video shot. Specifically, in a video shot that has a static background scene and dynamic foreground objects, the foreground objects in certain frames may occlude some portions of the background. These occluded portions may be exposed in other frames when the foreground objects move in front of the background. A background mosaic combines the background scenes exposed in every frame. The quality of a background mosaic is considered to be sufficient if the mosaic includes each part of the background scene ever exposed in the shot or if foreground objects have been completely deleted from the mosaic.
Once a background mosaic is constructed, it can be used to segment each video frame into background and foreground regions by comparing the intensities of the corresponding pixels in the background mosaic and the video frame.
Video frame segmentation, and background mosaicking itself, can be used in many applications. One application is video compression. A video may be stored or transmitted in the form of a background mosaic together with frames of foreground regions. Compared with the transmission of every frame, the mosaic/foreground form may save significant storage space and reduce transmitting bandwidth. The background mosaic may also contribute to machine content understanding. For example, a “clean” background, without the foreground objects that may confuse the system, provides better indication of where the video was shot. The background mosaic may also facilitate video-editing for home or studio use.